Skip to content

fix(loop-metrics): stop successRatePct from going negative on multi-escalation runs - #537

Open
KhaiTrang1995 wants to merge 1 commit into
cobusgreyling:mainfrom
KhaiTrang1995:fix/loop-metrics-success-rate-negative
Open

fix(loop-metrics): stop successRatePct from going negative on multi-escalation runs#537
KhaiTrang1995 wants to merge 1 commit into
cobusgreyling:mainfrom
KhaiTrang1995:fix/loop-metrics-success-rate-negative

Conversation

@KhaiTrang1995

Copy link
Copy Markdown
Contributor

Problem

successRatePct was computed as ((totalRuns - totalEscalations) /
totalRuns) * 100. totalEscalations is a sum of each run's escalation
count (a run can log more than one -- e.g. several items escalated
in a single triage pass; this repo's own loop-run-log.md has real
entries with escalations: 4 and escalations: 5), not a count of
runs-that-escalated. Subtracting an event count from a run count
conflates the two: any window where totalEscalations exceeds
totalRuns (trivially reachable with real data) produces a negative
percentage, which the CLI dashboard prints and color-codes as-is with
no clamping.

Fix

Success rate should mean "what fraction of runs didn't escalate at
all." Count runs with escalations === 0 instead of subtracting the
raw event sum, so the result stays correctly bounded to [0, 100]
regardless of how many escalations any single run logged.
totalEscalations itself is unchanged since roiScore correctly treats
each individual escalation event as a cost.

Test plan

Added a regression test with one run logging escalations: 5 and one
clean run, asserting successRatePct is 50 (not -150). Confirmed it
fails with the exact -150 result against the old formula and passes
with the fix. Full clean rebuild + npm test: 3/3 passing.

…scalation runs

successRatePct was computed as ((totalRuns - totalEscalations) /
totalRuns) * 100. totalEscalations is a sum of each run's escalation
*count* (a run can log more than one -- e.g. several items escalated
in a single triage pass; this repo's own loop-run-log.md has real
entries with escalations: 4 and escalations: 5), not a count of
runs-that-escalated. Subtracting an event count from a run count
conflates the two: any window where totalEscalations exceeds
totalRuns (trivially reachable with real data, e.g. two runs where
one logs escalations: 5) produces a negative percentage, which the
CLI dashboard prints and color-codes as-is with no clamping.

Success rate should mean "what fraction of runs didn't escalate at
all." Count runs with escalations === 0 instead of subtracting the
raw event sum, so the result stays correctly bounded to [0, 100]
regardless of how many escalations any single run logged.
totalEscalations itself is unchanged (still a raw sum) since roiScore
correctly treats each individual escalation event as a cost.

Test plan: added a regression test with one run logging
escalations: 5 and one clean run, asserting successRatePct is 50 (not
-150). Confirmed it fails with the exact -150 result against the old
formula and passes with the fix. Full clean rebuild + npm test: 3/3
passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@cobusgreyling

Copy link
Copy Markdown
Owner

Needs review: validate/audit are green, but this changes how successRatePct is defined (run-level “no escalations” vs subtracting total escalation events). That is a metrics semantics change, not a trivial fix — not merging from housekeeping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants